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THE NATIONAL CENTER MISSION STATEMENT 



The National Center for Research in Vocational Education's mission 
is to increase the ability of diverse agencies, institutions, and organi- 
zations .to solve educational problems relating to individual qareef 
planning, preparation, and progression. ThQ.National Center fulfills 
its mission by: , K 

• ■ * . 

* Generating knowledge through research 
Developing educational programs and products 
Fvnluatinq individual program needs and outcomes/ 
< * Installing educational programs and products * 
Operating information systems and services 
Condi^ing leadership deveiopmemjand training program* 



FOREWORD 



Program improvement in vocational education calls for timely investments in research and 
development activities in response to identified needs. Key elements in this complex process must 
be carefully orchestrated for efficient and effective use of resources. One state's approach to the* 
development and operation of a research coordinating unit is described in this paper. This model 
depicts program improvement as a continuous process involving such activities as priority determi- 
nation, research, product development and testing, diffusion, implementation, and impact assess- 
ment. Although this model may not fit into the management framework of every state, the key 
elements should be found in each state's program improvement operations. The author, Ronald D. 
McCage, has also provided a useful interpretation of existing legislative mandates in vocational , 
education. This one state's response to legislative mandates represents a valuable contribution to 
our knowledge of program improvement processes. 

,The concepts in this summary have been excerpted from a more extensive technical paper. 
The pgper contains addit tonal information on the legislative mandates, program improvement 
continuum, guides for writing and evaluating proposals, and final report/abstract formats. The 
technical paper has been submitted to the Educational Resources Information Center (ERIC) under 
the same title as this summary. 

/ We appreciate reviews of the draft document by Charles Hopkins, Assistant Director for 
Supportive Services, Oklahoma State Department of Vocational Technical Education; Charles 
Mcflkowski, Educational Consulting Services; and Floyd McKinney, Program Director, National 
Center. We are indebted to Brenda MacKgy and William Hull for their editorial assistance and 
advice provided in the development of this paper * . •* 



Robert Taylor 
Executive Director 
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MANAGING PROGRAM IMPROVEMENT; 
ONE STATE'S APPROACH 

This executive summary highlights key elements of a\aper produced by the National Center 
for Research in Vocational Education for persons interested In maximizing benefits from research 
ancTdevel«pment in vocational education. This paper describes the legislative background, conceptual 
model, essential framework and operational, guidelines for a state-managed vocational education 
program improvement system. One state, Illinois, is used to illustrate the program improvement 
function through the Research coordinating unit. The concepts in the paper may take various forms 
of relationships and procedures within a particular state governance structure* but the concepts of 
pnonty development, research and planning, development and testing, and diffusion/implementation 
are considered essential to a comprehensive program improvement framework. 

Legislative Background 

Legislation for the support of vocational education is a good example of Congressional concern 
for solving problems. In fact, vocational education/legislation has been one of the major vehicle 
used by Congress to address new and emerging social issues. TheSmith-Hughes Act of 1917 utilized 
vocational education as a method of retraining and rehabilitating the general population as well as 
veterans returning from World War I. The Vocational Education Act of 1963 encouraged the funding 
of training programs based on serving the individual needs of people as opposed to the prior empha- 
sis which provided funds for the support of selected categorical programs. In this Act Congress 
provided funds for a research coordinating unit (RCU) in each state to conduct research and develop- 
ment activities to improve and upgrade programs in the field. 

In 1967, the implementation of the Vocational Education Act was reviewed and evaluated 
Congress was displeased to learn that the social- issues implied in the 1963 Act had not been 
-addressed to the degree intended. The same basic programsTiad continued to receive support in a 
manner similar to the pre- 1963 era. One bright spot in.the Congressional review was the successful 
implementation of the research coordinating unit concept. Consequently/in passing the 1968 
Amendments to the 1963 Vocational Education Act, Congress made the RCU a permanent part of 
the states' program for vocational education program improvement. In addition, Congress also 
included line item categories for curriculum development and exemplary programs. 

Prior to amending vocational education legislation in 1976, the Congress conducted extensive 
oversight hearings to determine the strengths and weaknesses of the 1968 provisions. Simultaneously 
no less titan four major studies were conducted to determine the effect ivess of the $250 (XX) 000 
that had beep expended for research and development activities at the state and federal levels since 
i yob. t , 

The most significant of these studies was the two-year study by the Committee on Vocational 
Education Research and Development (COVERD) under the auspices of the National Academy of 
Sciences. Major COVERD conclusions and recommendations were parallel f* the general provisions 
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contained in Subpart 3, Program Improvement and Supportive Services of the EBucation Amend- 

ments of 1976, PL 94-482. '. ' ■ ■ . ♦ 

.< , . 

• Significant findings of the COVERD committee, which include but are not limited to the 

following, are: * • 

0 

1 CO VERD included all aspects of Part C- Research (federal and state shares), Part D- Exemplary 
. and Innovative Programs (federal and state shares) , and Part l-Curriculum Development 

(federal only) in its definition of research and development. COVERD charged that these 
programs had been conducted in isolation from each other, they had been operated withbut a 
sound set of long-range priorities, and no comprehensive model or plan was used for their 
management. Public Law 94-482 responded by eliminating the categorical line items in the 
1968 Amendments and by consolidating research, exemplary and-innovative programs, 
curriculum development, guidance and counseling, personnel development, arid grants for 
elimination of sex bias and sex role stereotyping under Subpart 3-Program Improvement and 
• Supportive Services. The Act further specified that program improvement, which was legis- 
latively defined as research, exemplary and innovative programs, and curriculum development, 
would be coordinated by a research coordinating unit. At a minimum, the Comprehensive 
Plan for Program Improvement would" contain the structure of the RCU, the state's priorities 
" for program improvement, the state's method' for addressing trie priorities, the stage's allocation 
of resources to each priority, and the procedures to be used^for dissemination of its products - 
and services to the local level. * 

2 COVERD charged that the $250,0O0;0O0 expended for research and development during the 
ten-year period (1965- 1975) of the 1963 Act and the % 1968 Amendments had no documented 
evidence of significant impact at the classroom level. Visible evidence was available, according 
to COVERD, but researchers and research managers had not done an adequate job of address- 
ing it The 1976 legislation addressed this problem by putting in two provisions which were 
aimed at providing greater cpntrdTand accountability. The first of these provisions was the 
need to use contracts to* accomplish objectives or to do work internally. Th/is provision applied 
to both the state and federal levels. Even though the originaj intent was admirable, the resulting 

. . problems in implementation have made the '-'contract only" form of funding less than desirable 
at both levels of government. The second measure passed by Congress was the inclusion of an 
impact or use statement within the law that put certain restrictions on research and curriculum 
activities. This provision requires the applicant to be able to show at the proposal stage that 
the idea put forth has a reasonable probability for significant use in a substantial number of 
classrooms or learning situations within five years of the termination of the contract. The 
ambiguity of this statement and' the lack of clarification in the Rules and Regulations has 
caused this provision to be less than effective. -Its inclusion of the words "curriculum materials" i 
and "instructional techniques" has caused most states to limit research activities to a curriculum 
thrust. " ' 

The Program Improvement Conceptual Moddl 

Illinois has developed a method of comprehensive program improvement to respond to 
national priorities. This process is based on a continuum which includes research, development, 
and disseminaion/diffusion activities designed to impact on local Jevel programming. This : V 
continuum consists of four major phases: 
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(1 } Priority development phase 

(2) Research and planning phase 

3) Development and refinement phase 

(4) Dissemination phase / 



ERIC 



The continuum is cyclical in function in that information and feedback obtained at each phase can 
result in revision and refinement of ongoing activities as well as provide direction to future activities. 
Inevitably, the feedback obtained through the dissemination phase (Phase 4) provides fdr the 
identification of new priorities to be addressed in Phase 1. This cyclical.process, described in 
figure 1, can result in a continuing responsiveness of vocational education to the emerging priorities 
of local educators. 



Priority Development Phase 

PL 94-482 requires that all program improvement activities be based on sound priorities. It is 
possible to determine and respond to national and state priorities through a process that is generated 
fronXstate'and local data. In Illinois, for exarriple, an abbreviated survey is distributed'annually to 
several diverse populations: educational administrators; vocational directors and classroom teachers 
at the elementary, secondary, postsecondary, and university level; key administrators and staff in 
business, industry, and labor; and private firms in educational management research and materials 
development. The information obtained from the survey is supplemented, clarified, and refined I 
through the^ examination of available data, review of related research, inquiries wjth state office 
staff, and input fronrconferences and meetings. The resulting priority listin^tends to represent the 
real program improvement needs at the local level that can be addressed by a state agency. 

% 

Research and Planning Phase 

During this phase the state should verify that a priority is in fact a real need. This phase 
should center around traditional research, status assessments, evaluations; feasibility studies, and 
activities that are designed to.detqrmine the state of the art in a given area. Major planning and 
design efforts should be conducted toward the end of this- phase that are based on the i^uicqjmes of 
the assessments and evaluations. These designs and planning efforts should serve as / *hebasis!or\ 
products or services to be developed and delivered in later phases. Diffusion strategies should be 
identified in this phase that will help'define selected activities in the development and testing phj 

Development and Testing Phase 

- The development and testing phase should be dedicated to those activities necessaryto produce 
a product and to provide a servjce? such as curriculum materials, program guides, instructional 
materials, arid supportive materials for teachers, counselors, and administrators. Illinois includes 
developmental wo/k toward inservice and preservice programs' as a part of this phase. Exemplary, 
and innovative programs can be demonstrated during this phase as one method of validating a 
concept or product v 

. - r v • 
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FIGURE 1. 

Program improvement Continuum Tor a Vocational 
Education Research and Development Operation 
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Concept features open entry/ open exit - Not everything appropriate to aJJ phases 
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PRIORITY 

DEVELbPMENT 

PHASE 



Input from: 

— Surveys 

— Conferences 

— Staff 

— Prior research 
reports 

— State Advisory 
Council on Adult, 
Vocational; and 
Technical Education 

— University Liaison 
Council 

— Annual Composite 
Evaluation Report 

— Industry 

— Business 

— Labor 



OUTPUT FOR 
RESEARCH AND 
PLANNING Pt^ASE 



Typical~Activities: 

- Research and 
design 

- Assessments 

- Evaluations 

- Feasibility 
studies 

- Planning and 
design 

- Review of 
literature 

- State of the 
art studies 



BASIS FOR 
DEVELOPMENT AND 
TESTING f>HASE 



TypicaU Outcomes: 

— Curriculum guides 

— Curriculum materials 

— Instructional 
materials 0 

— Inservice work- 
shops models 

— Preservice program 
. models . 

— Innovative models 

— Concept models 

— Administrative 
package models 

— Intern program 
models 

— Leadership pro- 
gram models 



CONTENT FOR 
DIFFUSION AND 
IMPLEMENTATION 
PHASE 



Typical Vehicles: 

— Reports and materials 
dissemination 

— Demonstration 
center 

— Inservice workshops 

- Preservice programs 

— Administration ' 
internships , 
Exchange programs 

- diffusion networks- 
/Dissemination 

conferences 

- Curriculum 
management center 

- Publication 
clearinghouse 




Diffusion and Implementation Phase 



"Techniques used during the diffusion and implementation phase have the ultimate goal of 
implementation in the classroom or other learning situation. The tasks, may be simple or complex, 
such as mailing, a document or conducting inservice workshops, preservice programs, internships, 
exchanges, or demonstration centers. The major purpose of the diffusion phase is to provide con- 
cepts and materials to the local level with the necessary technical assistance to implement the 
innovation. Simultaneous preservice or inservice is necessary for internalization of the innovative 
concepts by staff. j 

f Essential Framework , , 

A. Background * 

In order to prepare adequate guidelines for developing a framework, the following assumptions 
and parameters have been established: . 
i ' ' 

1. The Rules and Regulations for implementing Subpart 3, Title II bf the Education Amend- 
ments of 1976 indicate that if a state chooses to conduct program improvement activities, 
which i nc I ude~ Section 131 (Research), Section 132 (Exemplary and Innovative Programs), 
and Section 133 (Curriculum Development), they' must be coordinated through the research 
coordinating unit (RGU). Since the Research and Development Section of the Department 
of Adult, Vocational, and Technical Education/Illinois State Board of Education has been 
designated as the Illinois RCU, this section is responsible for preparing a comprehensive plan 
for program improvement as part of the state's One- and Five- Year Plan. This comprehensive 
plan must include: ^ 

* a Priorities for program improvement. / • - 

b. Methods to address these priorities \ 
c Allocation of resources for each priority 
d Dissemination procedures ' • 
e. Impact indicators ' 

2. Section 134 (Vocational Guidance and Counseling), Section 135 (Vocational Educational 
Personnel Training), and Section 136 (Grants to Assist in Overcoming Sex Bias) are grouped 
under th£ umbrella of Supportive Services. Legislatively there are no ties between these t 
funds and the state RCU function, but these activities can be managed by the RCU if the 
state chooses. 

3. fn Illinois, administrative responsibility for Sections 131, 132, 133, 135, and the sex equity 
grants provision of Section* 136 has been assigned to the RCU. Placing the funding \fcith this 
section provides for utilization of already existing processes for handling grants and contractual 
funding agreements. 

» * • . «. " 

4. The State Board of Education requires the following items in funding agreements: 

a. Measurable objectives 4 
% b. Time-specific and outcome-oriented procedures related to each objective 

c. Formative and suramative evaluation processes tied to each objective and/or 
r procedure that places a major emphasis on impact 

d. Dissemination plan ^ 

e. Designation of job descriptions, namesL and qualifications^ personnel 

f. Designation of deliverables 

g. 'Program budget • ' 

h. Abstracts for summary and reporting requirements 
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• The process for developing contracts follows these steps: 

a. Pre-established priorities 

* b. Issuance of requests for proposals (RFPs) 

c. Competitive review t J 

d. Award of a funding agreement with predetermined and prespecified outcomes. 

When a contract is based on an RFP, the board contracts with an external agency to 
perform a service or develop a product. Most of the tasks and methods used to address the 
issue are part of the funding agreement, and any changes made in the tasks or methods 
must be accomplished through negotiations between the -board and the contracted agency. 

5. The Research and Development Section has been given responsibility for contracting and 
establishing rules and regulations for this process. The contract administrator within this 
section' is responsible for coordinating the contracting process and insuring that all manage- 
ment and monitoring functions are fulfilled. This staff person is held responsible ior the 

. success or failure of the given activity. j 

B. Project Categories 

1. A number of projects funded from Section 130 *o not require a content expert/resource 
specialist dn an ongoing basis. For these projects, the contract administrator does everyth 
necessary to fulfill the requirements of management and monitoring. 

* 

2. Increasingly the Research and Development Section is funding projects that fall under the 
broad category of program improvement and deal specifically with the development of 
curriculum and/or instructional materials used in local schools. This category of projects 
places the highest demand on the board for subject matter expertise from other sections 
and/or outside the department. The need for this involvement is based on such considerations 
as type of project, stage of development, and subject matter. 

' 3. The third category of projects is funded from other than Section 130 and involves the 

support and operation of programs at the local level. The Research and Development Section 
is responsibly for the project's fiscal management, while the program management remains 
with the section originally responsible for the management of tfcat source of funds. 

4. The fourth category includes activities, such as career guidance centers, funded from 
Section 130 for which the Research and Development Section is given budget preparation 
responsibility, while the overall management remains with the Special' Program Section. 

C. Framework 

Fr.om the perspective of the Research'and Development Section, there are four major points 
in the program improvement process that require the involvement of more than one section. * 
Specific procedures for interface have been determined for each of the four stages. 
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1. Priority Input and Development • f • m 

a. In August the Research and Development Section initiates the priority determinatipn ^ 
process by conducting surveys and information analysis. ✓ % 
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b. By October 30 all sections of the Department of Adult, Vocational and Technical Education 
are asked to submit prioritized needs to the Research and Development Sectiof* along with 
needs input from the field and other sources. ^ # V" 

c, By November 15 the section staff analyzes and summarizes ail data and returns a 
composite to all other section managers for review and comment 

± By December 1 all input is returned to the section for analysis and consolidation. 

e. By the second Monday in December the section staff prepares a list of priorities with 
documentation and presents this list at a formal meeting of the Administrative Council 

for acceptance and approval. 

*- » 

f. Given final approval, the Research and Development staff translates the priorities into 
activity statements for inclusion in the annual plan. 

2. Request for Proposal (RFP) Development 

a. By January 15 the Research and Development Section manager meets with other section 
managers to determine the assignments for writing requests for proposals (R£Ps), These 
assignments are generated from topics appearing on the approved priority list 

b. By January 30 a contract administrator and a resource person are assigned to each activity 
with the exception of open category activities. These cannot be done at this tirrie due to* r $ 
the nonspecific nature of the open category concept - > 

c. The section makes all final arrangements for preparation and issuance of the RFP booklets 
by March 15. 

3. Proposal Review j ^ 

a. By February 15 the Research and Development Section manager meets with the managers 
of other sections to discuss the need for personnel to review proposals. 

b. By March 1 the managers having personnel involved in research and development project 
reviews assign reviewers from within their section in accordance with the categorical need 
agreed upon by ell managers involved. In addition, section managers are asked to nominate 
people from the field who would make good proposal reviewers: 4 

c. Wher^possibie, it expedites the process if the same person originally designated as a 
writing resource p^fson for £ task-specific RFP is assigned to review and later monitor 
the project in question. This assignment philosophy is consistent with the Research and 
Development Section practice of a contract administrator following a project from incep- 
tion to comflletion. <~ 

4. Project Monitoring 

a. Once a contract administrator and a content specialist h&e been named, these individuals 
meet to plot the course of a given activity. Plotting the course is based on the RFP and 
the irjtent of the activity. Role and ft/hction of each person is delineated at this time. 

b. Once a proposal -has been selected for funding, the contract administrator and the resource 
person meet to agree on points to be negotiated. Once these are agreed upon, the contract 
administrator proceeds with the negotiations. Disputes are resolved tia the chain of 
command prior to entering negotiations. * ' 
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c. Within 30 days after a contract has been approved, the contract administrator ari^i the 
resource person meet with the contracted agency to lay out a plan of action. - ' 

d. Periodically throughout the contract, meetings are held in accordance with the finding 
agreementand the^greed-upon plan of action.' in every case- the contract adminJetfator 

* . codrdinates with other staff concerning the agenda for these meetinqs 

. • i * . / • . 

e. Project personnel are advised of the relationship between internal personnel and their 
role in the funded activity.' In all cases, unless otherwise agreed upon, the project directors 
and staff are advised that Jheir official contact of the board is the contract administrator ' 

for that project. . . 
• ■ « . 

This'^rocess is summarised in figure 2. 

^ " v 

Guidelines for the Operation of a 
Program Improvement System 

• Obviously, there are several elements essential to the effective development and operation of * 
a-'comprehensive progfcim improvement system for vocational education through a research coordi- 
nating unit. The guidelines contained on the following pages are presented to states interested in 
developing and/or upgrading the program improvement function. The guidelines are presented in a 
brief and somewhat concise form so that they can be used as a review checklist. There are some 
key assumptions necessary to the understanding of these guidelines. It is assumed that the reader 
does have or will gain: . «' 

1. 9 A thorough knowledgeof ail current legisiation^and rules and regulations that* are 

applicable to the program improvement/research coordinating unit function in 
vocational education. 

2. A knowledge of existing literature that is relevant to the management and operation of 
a research coordinating unit. The RCU should be an integral part of the state vocational 
education structure. 

3. A philosophical or conceptualunderstanding of the program improvement function 
within 3 vocational education framework, 

. 4. A conceptual understanding of the purpose of a research goordinating unit. 

Given these assumptions, the following guidelines Sre absolutely essential to the development and 
operation of a comprehensive system of program improvement in vocational education: 
* ■ 

1. Gain or have the administrative support of the state director of vocational education for the 
concept of program improvement. Without administrative support and invoivement'of key 

" staff, the concept -will fail. ^ 

r 

2. A single unit should be responsible for coordinating all elements of the program improvement 
system. The current legislation calls this a research coordinating unit but this term may b£ 
outdated. 

3. A state should h^ve a conceptual base or operational modei, such as research and assessment, 
product/service development and testing, diffusion and personnel development, etc. with 
assessment and evaluation mechanisms built into the operational model. This mode! provides 
for the articulation and coordination in and among the elements. 

8 
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FIGURE 2. 

Task /Time Framework for 
State Program Improvement Activities 



TASK 



Aug. Sept. Oct. Nov. Dec. Jan. fW. Mar. Apr? May June July 



July 



S. 
s 7 

f 

> 

O 3 

C 

a 
c 5 



Prepare and disseminate needs 
assessment survey 

Receive .and analyze siyveys and 
other needs assessment data 

'Prepare composite of data and sub- 
mit to section managers for review 

Section managers return data 

Prepare listing and documentation 
of priorities for approval 



.O 6. Translate priorities into activity 
ol statement for inclusion in the 
annual plan 



g 7. Determine assignments for 

£ RFP writing 

a 

-5 8. Assign contrac^admimstrator 
J ^>*nd resource person to each 
Q . RFP activity 

a. 

^ 9. Prepare and issue RFP book 



10. Receive proposals 



g 1 1 /Assign proposal reviewers 
^ 12. Review proposals 
Q-13, Negotiate proposals 
14. Sign proposals into contracts 



15. Prepare action plan 

03 
c 

'w 16. Monitor contracts 
O 

c 17. Negotiate any contract 
5 revisions 

^ 18. Receive and process claims 
5* for reimbursement 

19. Receive products and 
final reports 



Legend 



Planned schedule 
Tentative schedule 
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Define program improvement as a comprehensive research, development, and diffusion 
process that is driven by a priority or state/local needs base as opposed to a collection of 
categorical programs such as research, curriculum, etc. 

r 

Have a comprehensive plan for program improvement which includes the administrative 
structure of the RCU, the annual priorities for program improvement activities, a description ; 
of the method for addressing these priorities, the allocation of resources by priority or need ■ 
and the various techniques to be used far dissemination and diffusion of- the products as well 
as the methods to be used for accountability and input. Have a plan ! Work the plan! 

6. develop a needs assessment process that produces sound data for identification of priorities 

- ;•«> • for program improvement activities that looks at all aspects of vocational education and places 
a high level of emphasis on input and use of information from the practitioner at the local 
level as well as emphasis on traditional sources such as reports, advisory groups, and state staff. 

7. Given a legislative directive for doing work internally or contracting externally, develop an 

» equitable and fair process for soliciting and identifying those agencies that will address the . 

\ state's priorities. Probably the most common competitive process is the request for proposa'l 

(RFP) process. Regardless of the process, insure that everyone involved knows the rules and 
that the sysfem \£ operated according to the rdjes. An above-the-board, honestly operated 
system will generally be accepted and supported by the field. 

8. Develop a method for evaluating activities while in progress and after completion. This 
process needs to be highly formalized so that the research coordinating unit is provided-with 
accountability data for long-range impact purposes. Field testing and validation techniques 

should be an integral part of each activity. This evaluation and impact system should be inte- .* 
grated into every activity of the unit on an ongoing basis. 

> . 

9. Dissemination and diffusion activities should be planned into every activity from the beginning 
of the activity. Without a good diffusion and dissemination process, one cannot expect to 
document significant impact referenced in the legislation. Make maximum use of the developers 
in the diffusion process. They know their product best. 

# 

* 10 - Personnel development activities, both preservice and inservice, should be considered integral 

to a total program improvement concept. Not all personnel development activities can be * 
defined as diffusion techniques, but preservice and inservice techniques can certainly be used 
effectively as diffusion devices. 

11. Have an effective activity /project monitoring system that maximizes use of available staff and 
resources. Be sure the system is accountable within itself. Be sure contractual documents are 
outcome-based and budgeted in such a way that financial accountability can be established 
through auditing. 

12. Have a system for the integration of other agency staff into the program improvement process. 
Maximjze the use of resource people in their area of expertise using RCU staff as facilitators 
of the process. " . ; 

* " 

13. Use local people to the greatest extent possible. They are where the action is. They are on the 
cutting edge and know the problems first.- They are al|p the best barometers of whether 
something will work or not. 
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14.^ Have a well-balanced staff in terms of expertise, equity, etc. Assign responsibilities to staff and 
let them operate. \' ; 

.15. Be a risk taker. Nothing innovative will ever occur unless you are willing to stick your" neck out. 
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